maint: vram-first validation — probe override, record-and-flag pins (polish phase 2)#55
Merged
Merged
Conversation
…ce (#54) - vram/config.py: VMAP_BATCH_SPARSE table (w-tilde rows; unprobed cells fall back dense), PROVENANCE record (dense table probed 2026-05-24 — stale vs current source, re-probe submits staged), and resolve_vmap_batch(): runtime cells now prefer a fresh cell-matching vmap_probe_<model>[_sparse].json over the static table, so a stale table cannot OOM a job whose submit re-probed in Phase A - all 8 vmap-capable runtime cells wired to resolve_vmap_batch with the winning source logged; probe JSON basenames normalised to vmap_probe_<model>[_sparse].json across interferometer/point_source/ datacube (imaging cells already followed it) - hpc/batch_gpu/submit_probe_{fast,delaunay}_a100: probe-only sweeps re-validating every campaign vmap cell (dense + sparse imaging) against current source; ingest workflow documented in vram/README.md, which also loses its stale z_projects path - test/test_vram_config.py: 11 new tests (sparse fallback, probe override, mismatch/corrupt-JSON handling) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Design decision (user-directed): autolens_profiling's pinned values guard benchmark comparability, not library correctness — that remit belongs to autolens_workspace_test. A hard assert conflated "computation changed" with "profiling run failed" and could kill a multi-hour A100 job at the final assertion. - _profile_cli.check_pinned(): compare against the pin, warn loudly on drift, return a drift record; record_pinned_check() merges pinned_expected + pinned_drift into the already-written result JSON — the machine surface PyAutoHeart's vitals scan will read (feature/pyautoheart/profiling_drift_check.md filed) - all 9 runtime cells converted; eager ≡ JIT ≡ vmap cross-consistency assertions stay hard (they invalidate the measurement itself) - imaging mge/pixelization/delaunay pins made per-instrument first (the old module-level HST constants were asserted for every --instrument, failing any non-hst run by construction — flushed by this sweep) - vram: non-GPU backends clamp resolved batches to 3 (A100 table applied on laptop CPU OOM-killed the sweep); probes record their backend and resolution ignores probes from a different backend; +12 tests (25/25) - boundary rule recorded in results/notes/design_lock_in.md Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
All 9 campaign cells green on CPU under the fixed code (results/notes/ vram_validation_2026_07_08.md is the durable record — sweep artifacts were scratch-tier). Simulators dashboard row for the auto-simulated euclid dataset (v2026.7.6.649). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Two lint-gate failures, one root cause each: - PR gate: build_readme --check diverged because results/simulators/ was still gitignored (a pre-phase-1 "until we decide" rule) — the local dashboard baked a euclid row from an artifact CI never received. The phase-1 design settled that results live in-repo: un-ignore results/simulators/ + results/searches/ (results/likelihood/ stays ignored as the retired legacy section) and commit the euclid artifact pair. - main gate (lychee): simulators/README.md still linked the retired likelihood/README.md; retargeted to likelihood_runtime/breakdown. Repo-wide relative-link scan: no other dead links. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…me/breakdown split; exposed once lychee passed)
…picked pip list lacked pyyaml etc; adopt the lib-tests.yml pattern
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 2 of the polish series (closes the local leg of #54): validate the VRAM/vmap story before the PreOptimizationTimes campaign burns A100 hours. All 9 campaign cells now run green end-to-end on CPU (durable record:
results/notes/vram_validation_2026_07_08.md); the sweep flushed three real defects, all fixed here. The A100 probe leg is staged (submits + ingest workflow) for when RAL is back.Scripts Changed
vram/config.py—resolve_vmap_batch(): runtime cells prefer a fresh cell-matchingvmap_probe_<model>[_sparse].jsonover the static table (a stale table can no longer OOM a job whose submit re-probed);VMAP_BATCH_SPARSE(w-tilde rows, unprobed cells fall back to the dense value);PROVENANCE(dense table dates to 2026-05-24 — recorded explicitly); non-GPU backends clamp resolved batches to 3 (the A100 table OOM-killed laptop-CPU runs at batch 64)vram/probe.py— probe JSONs record the backend that produced them; resolution ignores probes from a different backend (CPU memory analysis says nothing about A100)_profile_cli.py—check_pinned()/record_pinned_check(): pinned likelihood/evidence drift warns and records (pinned_expected+pinned_driftin the result JSON — the surface PyAutoHeart's filed vitals leg scans) instead of crashing a multi-hour job; boundary rule inresults/notes/design_lock_in.md(profiling records and flags; autolens_workspace_test adjudicates library correctness)likelihood_runtime/<class>/<model>.py(all 9 cells) — wired toresolve_vmap_batchwith the winning source logged; pins converted to record-and-flag (eager ≡ JIT ≡ vmap cross-consistency asserts stay hard);imaging/{mge,pixelization,delaunay}pins made per-instrument (the old module-level HST constants failed any non-HST run by construction); probe basenames normalised tovmap_probe_<model>[_sparse].jsonhpc/batch_gpu/submit_probe_{fast,delaunay}_a100— probe-only sweeps re-validating every campaign vmap cell (dense + sparse imaging) against current source; ingest workflow invram/README.mdtest/test_vram_config.py— 17 new tests (sparse fallback, probe override, mismatch/corrupt/backend handling); 25/25 totalresults/notes/vram_validation_2026_07_08.md— sweep verdict, timings, defects;dataset/imaging/euclid/+ simulators dashboard row (auto-simulated during the sweep)Test Plan
ruff check+format --checkclean;build_readme.py --checkidempotentValidation checklist (--auto run — plan approved in-session)
🤖 Generated with Claude Code